Corruption of Generalizing Signals in Densely Connected Feedforward Neural Networks with Hyperbolic Tangent Activation Functions

This paper discusses the propagation of signals in generic densely connected multilayered feedforward neural networks. It is concluded that the dense connecting combined with the hyperbolic tangent activation functions of the neurons may cause a highly random, spurious generalization, that decreases the overall performance and reliability of a neural network and can be mistaken with overfitting. Modified activation functions instead of the hyperbolic tangent activation functions and an organized, instead of ad hoc, way of connecting neurons are discussed as possible ways of reducing the spurious generalization. keywords: feedforward neural networks, generalization, overfitting